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Background / Context: 

A central issue in nonexperimental studies is the identification of comparable individuals 
(e.g. students) to remove selection bias. One such increasingly common method to identify 
comparable individuals and address selection bias is the propensity score (PS). PS methods rely 
on a model of the treatment assignment to identify comparable individuals on the basis of similar 
probabilities of receiving treatment. To extend PS methods to education data, the methods must 
be appropriately adapted to address the potential multilevel nature of treatment assignments. For 
instance, in studies that resemble multi-site randomized trials, treatments are assigned to students 
within each school. Such assignments are often based on student and school characteristics while 
the mechanism of assignment may vary substantially across schools. Correspondingly, selection 
bias may originate from both student and school levels. One recent examination of this issue 
includes estimating the effect of kindergarten retention on achievement where students' 
probability of being retained is thought to be a function of the student's personal characteristics 
but also his/her school membership (Hong & Raudenbush, 2006). 

To attend to how the treatment seleetion mechanism differs among sehools, two principal 
approaches have been proposed. The first approach makes use of a canonical single level PS in 
conjunction with restricting comparisons to individuals within the same school (Rosenbaum, 
1986). This approaeh addresses the varying influenee of a school by blocking on school 
membership, and thus on observed and unobserved school covariates. A second approach to 
address varying sehool influence on treatment assignment is use of multilevel PSs (Hong & 
Raudenbush, 2006; Kim & Seltzer, 2007). This approach explicitly models how treatment 
assignment mechanisms differ among schools potentially making PSs comparable across school 
boundaries. In using sueh approaehes to identify comparable students and estimate effeets, both 
approaches hold attractive features and potential drawbacks. In particular, use of single level PSs 
which constrain comparisons to students within schools may alleviate or protect against residual 
bias from unmeasured school level factors that influence the treatment assignment. However, 
such protection is often coupled with a lack of comparable matches within each school. In 
particular, restricting comparisons to within schools generally requires a large reservoir of 
treatment and control students in each school to ensure high quality matches. In contrast, by 
taking into account both school membership and how the predictive capacities of covariates vary 
across schools, multilevel PSs can more effectively match students aeross schools. That is, 
because multilevel PSs take into account the extent to which school influence on student 
treatment assignment varies, estimated probabilities of receiving treatment are more comparable 
across schools. As a result, the reservoir of potential matehes is much larger and often leads to 
higher quality matches. Such advantages, however, are also paired with the assumption that there 
are no unmeasured school level covariates. Because multilevel PSs permit matches aeross 
schools, they assume adequate adjustment for the differences in the selection processes. 

Purpose / Objective / Research Question / Focus of Study: 

In this study, we evaluated the performance of both of these approaches in 
nonexperimental studies resembling multisite randomized trials. In particular, we assessed the 
performance of matching students across schools based on multilevel PSs and matching students 
within schools based on single level PSs with specific focus on the sensitivity of the PSs to 
omitted covariates. More specifically, we examined the extent to whieh matching across groups 
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using a multilevel PS produces a more effective estimator of the treatment effect as compared to 
matching within groups using a single level PS. We addressed the research question by assessing 
the performance of the treatment effect estimators in multiple scenarios using Monte Carlo 
simulation. For proposal brevity, we highlight four simple situations. 

1 . No unmeasured covariates that influence the treatment assignment 

2. One unmeasured school level covariate that influences the treatment assignment 

3. One unmeasured student level covariate that influences the treatment assignment 

4. One unmeasured school level covariate and one unmeasured student level covariate; both 
of which influence the treatment assignment 

Setting: 

(May not be applicable for Methods submissions) 

Population / Participants / Subjects: 

(May not be applicable for Methods submissions) 

Intervention / Program / Practice: 

(May not be applicable for Methods submissions) 

Significance / Novelty of study: 

Multilevel PSs potentially hold means to both remove selection bias associated with how 
selection processes differ among schools as well as from bias associated with omitted school 
level variables. Accordingly, multilevel PSs estimate students' probabilities of receiving 
treatment in a way that is both robust and more comparable across schools. In turn, such features 
frequently promote higher quality matches and tend to remove more selection bias. Although 
matching within schools lays claim to high face validity, the potential bias reduction gained by 
allowing across school matches has important implications. In particular, studies which examine 
treatments of low incidence (e.g. retention) are well positioned to gain from across school 
matching based on multilevel PSs. Similarly, matching across schools based on multilevel PSs in 
studies which examine treatments implemented at the teacher/classroom level may improve 
estimation. That is, because evaluating treatments implemented at the teacher/classroom level 
requires matching intact classrooms, the potential matches within a school (e.g. another 
classroom in that school and grade) can be very limited. In such situations, a method which 
affords researchers the ability to identify comparable classrooms in other schools may be highly 
beneficial and allow evaluation of treatments that otherwise could not have been evaluated. 

Statistical, Measurement, or Econometric Model: 

We performed four Monte Carlo simulations, each with 1,000 simulated data sets. The 
first simulation examined the performance of matching within and across schools based on both 
multilevel and single level PSs. The remaining simulations examined the approaches sensitivity 
to omitted variables at both the student and school levels. We generated the individual 
characteristics (A 7 W 2 W 5 ) and group characteristics (JVj, JV 2 , W 3 ) using independent normal 
distributions with mean 0 and variance 1. Sample sizes presented here focus on 100 schools with 
an average of 10 students per school. We designed the treatment as the realization of a 
dichotomous variable given student characteristics and school characteristics. For each 
simulation, the true treatment assignment mechanism followed the hierarchical generalized linear 
model: 
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Level 1 : logit(P(Z = 1)) = ^ 
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The true outcome model for each simulation was a random intercept and slope 
hierarchical linear model (HLM) 

W = 1 

Level 2 : /?o = + 2 y„oKj + % (0-3) 
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For proposal brevity we limit our discussion to simulations in which each coefficient in (0.1) 
equals ±0.3 whereas the coefficients in (0.3) equal ±0.5. 

To estimate each student's probability of receiving the treatment we constructed PSs 
using different versions of (0.1) and highlight six different PS specifications which constrain 
various parameters in (0.1). The specifications are: 

(1) Constrain all random effects {u) and cross level interactions {jnm) to be zero {Single level 
main fixed effects only) 

(2) Constrain all random effects {u) {Single level all fixed effects) 
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(3) Constrain slope random effects {umj) and cross level interactions (y„m) to be zero 
{Multilevel with a random intercept and main fixed effects only) 

(4) Constrain slope random effects {umj) to be zero {Multilevel with a random intercept and 
all fixed effects) 

(5) Constrain both school fixed effects {jno) and cross level interactions (y„m) to be zero 
{Multilevel with random intercept and slopes and only level one main fixed effects) 

(6) Do not constrain any parameters in (0.1) {Multilevel with random intercept and slopes 
and all fixed effects) 

To estimate the treatment effect for each data set and approach, we combined adjustment 
on the PS with a HLM (e.g. Hong & Raudenbush, 2006; e.g. Hirano & Imbens, 2002; Kleyman 
& Hansen, 2008). Here, we limit our discussion to matching students within or across schools 
using full matching so as to make use of all available students (e.g. Hansen, 2004). Using match 
indicators, we modeled the outcome as a HLM as follows: 

q=y tu.o; 

Level 2 = 7oo + % 

where <5 is the estimated treatment, M is the matched group, and Q is the number of matched 
groups minus one. To compare the estimates based on the different classes and specifications of 
models, we used the results of the Monte Carlo simulations to estimate the bias and mean- 
squared error (MSB) of each approach. We estimated these quantities using: 
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where N represents the number of simulated data sets. 

Research Design: 

(May not be applicable for Methods submissions) 

Data Collection and Analysis: 

(May not be applicable for Methods submissions) 

Findings / Results: 

In the first simulation we examined the quality of treatment effect estimates when all 
covariates were observed. The results suggested that estimators which allow matching students 
across schools outperform estimators that restrict matches to within schools (Table 1). This result 
was anticipated as the main utility of restricting matches to within schools to eliminate 
unobserved school level bias was irrelevant. As a result, matching across school boundaries 
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tended to promote higher quality matehes. Results from the first simulation additionally 
suggested that matehes based on multilevel PSs tend to outperform estimators whieh mateh 
students based on single level PSs regardless of whether matehes were within or aeross sehools 
(Table 1 & Tables not presented here). The seeond simulation eompared the performanee of the 
different approaehes when a sehool level eonfounding variable was omitted from the analyses. 
Evident from Table 2, the performanee of sehemes whieh matehed aeross sehools degraded 
whereas within sehools matehing sehemes did not (Table 2). However, those approaehes whieh 
matehed aeross sehools based on a multilevel PS still appreeiably outperformed those approaehes 
whieh matehed within sehools based on a single level PS (Table 2). Similar results held for 
seenarios whieh omitted a student and/or a sehool level eovariate (Tables 3 & 4). 

Usefulness / Applicability of Method: 

The results presented primarily suggest that even with omitted variables there is an 
appreeiable advantage to matehing aeross sehools when one employs multilevel PSs. To a large 
extent, PSs whieh use methods to address how seleetion proeesses differ among sehools (e.g. 
multilevel PSs) tended to be robust to sehool level omitted eonfounding variables. That is, 
beeause multilevel PSs employ random effeets, they ean both summarize omitted sehool level 
variables and aeeount for the aeross sehool variation in the predietive eapaeity of eovariates. As a 
result, multilevel PSs potentially hold means to both remove seleetion bias assoeiated with how 
seleetion proeesses differ among sehools as well as from bias assoeiated with omitted sehool 
level variables. 

Conclusions: 

In summary, the results of this study suggest that matehing aeross sehools based on 
multilevel PSs has mueh to offer and that further study into its utility and the eonditions under 
whieh it outperforms matehing within sehools based on a single level PS need to be explieated. 

In partieular, the simulations presented address small eovariate spaees with simple variable 
relations. Study of designs with more eomplex assoeiations among both unobserved and 
observed eonfounders is warranted. Finally, omitted in this summary are more eomplex details 
eoneerning eovariate balanee. Speeifieally, obtaining and assessing eovariate balanee both within 
and aeross sehools to build a strong basis for inferenee is mueh more eomplex. 
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Appendix B, Tables and Figures 

Table 1: Comparison of within school and across school matching using several different single 
and multilevel propensity scores approaches and assuming there are no omitted covariates 



Approach 

Matching 
PS Type 
Fixed 
Effects 
Random 
Effects 


(1) 

Within 

Single 

Main 

None 


(1) 

Across 

Single 

Main 

None 


(2) 

Within 

Single 

All 

None 


(2) 

Across 

Single 

All 

None 


(3) 

Across 

Multilevel 

Main 

Intercept 


(4) 

Across 

Multilevel 

All 

Intercept 


(5) 

Across 
Multilevel 
Level 1 
only 

Intercept & 
Slope 


(6) 

Across 

Multilevel 

All 

Intercept & 
Slope 


10 X Bias 


6.63 


5.06 


6.37 


0.89 


5.21 


-0.18 


-0.56 


-0.53 


10 X Var 


0.37 


0.40 


0.36 


0.28 


0.40 


0.29 


0.28 


0.29 


lOxMSE 


4.77 


2.96 


4.42 


0.35 


3.12 


0.29 


0.31 


0.32 



Note: We also examined, for instance, matching within schools based on multilevel PSs but do 
not present the results due to space limitations 
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Table 2: Comparison of within school and across school matching using several different single 
and multilevel propensity scores approaches and assuming there is one school level covariate 
that is omitted 



Approach 

Matching 
PS Type 
Fixed 
Effects‘‘ 
Random 
Effects 


(1) 

Within 

Single 

Main 

None 


(1) 

Across 

Single 

Main 

None 


(2) 

Within 

Single 

All 

None 


(2) 

Across 

Single 

All 

None 


( 3 ) 

Across 

Multilevel 

Main 

Intercept 


( 4 ) 

Across 

Multilevel 

All 

Intercept 


( 5 ) 

Across 
Multilevel 
Level 1 
only 

Intercept & 
Slope 


(6) 

Across 

Multilevel 

All 

Intercept & 
Slope 


10 X Bias 


6.57 


6.70 


6.65 


3.42 


6.43 


2.95 


-0.52 


-1.10 


10 X Var 


0.38 


0.43 


0.39 


0.35 


0.44 


0.36 


0.31 


0.31 


lOxMSE 


4.70 


4.92 


4.80 


1.52 


4.57 


1.23 


0.34 


0.43 



“All fixed effects in Table 2 exclude W 3 a school level covariate which predicts the treatment 
assignment 



Table 3: Comparison of within school and across school matching using several different single 
and multilevel propensity scores approaches and assuming there is one student level covariate 
that is omitted 



Approach 

Matching 
PS Type 
Fixed 
Effects’^ 
Random 
Effects 


(1) 

Within 

Single 

Main 

None 


(1) 

Across 

Single 

Main 

None 


(2) 

Within 

Single 

All 

None 


(2) 

Across 

Single 

All 

None 


( 3 ) 

Across 

Multilevel 

Main 

Intercept 


( 4 ) 

Across 

Multilevel 

All 

Intercept 


( 5 ) 

Across 
Multilevel 
Level 1 
only 

Intercept & 
Slope 


(6) 

Across 

Multilevel 

All 

Intercept & 
Slope 


10 X Bias 


7.82 


6.75 


7.68 


3.85 


6.84 


3.52 


3.76 


2.33 


10 X Var 


0.41 


0.44 


0.41 


0.35 


0.45 


0.35 


0.34 


0.36 


lOxMSE 


6.52 


4.99 


6.31 


1.82 


5.13 


1.59 


1.76 


0.90 



*A11 fixed effects in Table 3 exclude X 3 a student level covariate which predicts the treatment 
assignment 
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Table 4; Comparison of within school and across school matching using several different single 
and multilevel propensity scores approaches and assuming there is one student level and one 
school level covariate that is omitted 



Approach 

Matching 
PS Type 
Fixed 
Effects'^ 
Random 
Effects 


(1) 

Within 

Single 

Main 

None 


(1) 

Across 

Single 

Main 

None 


(2) 

Within 

Single 

All 

None 


(2) 

Across 

Single 

All 

None 


(3) 

Across 

Multilevel 

Main 

Intercept 


(4) 

Across 

Multilevel 

All 

Intercept 


(5) 

Across 
Multilevel 
Level 1 
only 

Intercept & 
Slope 


(6) 

Across 

Multilevel 

All 

Intercept & 
Slope 


10 X Bias 


7.99 


8.03 


7.98 


6.11 


1.11 


5.60 


3.88 


3.25 


10 X Var 


0.41 


0.45 


0.40 


0.37 


0.45 


0.38 


0.37 


0.35 


lOxMSE 


6.80 


6.89 


6.77 


4.10 


6.48 


3.51 


1.88 


1.41 



^All fixed effects in Table 4 exclude X5 a student level covariate and W 3 a school level covariate; 
both of which predict the treatment assignment 
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